-
Notifications
You must be signed in to change notification settings - Fork 803
Refactor CLI implementation using Typer #3365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very excited about this switch! I left some very high level comments on code organization :)
setup.py
Outdated
|
||
extras["cli"] = [ | ||
"InquirerPy==0.3.4", # Note: installs `prompt-toolkit` in the background | ||
"typer", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would move typer
to a required dependency otherwise doing pip install huggingface_hub
would install a CLI but unusable (since typer would be missing).
To reduce dependency overhead, especially the rich
package, let's add typer-slim
which is a lighter version without rich / shellingham included. More details here: https://typer.tiangolo.com/?h=typer+slim#typer-slim
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
About shellingham
I still think it's good to have it. Either by default or in the (soon-to-come?) "hf installer script". It's a pure-python package that helps detecting the current shell to install shell completion correctly.
(for now let's skip it but mentioning it to have it in mind)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agree! it's an oversight from me, fixed in 8ee6892
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks promising assuming commands implementations have not been changed^^
(I did not review the tests yet)
) -> None: | ||
logging.set_verbosity_info() | ||
repo_type = validate_repo_type(repo_type) | ||
api = HfApi(token=token, library_name="hf") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's actually a really good idea to track usage of our CLI. I think it'd be good to centralize this (some other HfApi
instantiations are not defining the library_name) but can be done in a later PR. (we have a subticket about analytics specifically)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
api = HfApi(token=token, library_name="hf") | |
api = HfApi(token=token, library_name="hf", library_version=__version__) |
might be nice to add huggingface_hub
version as well (would make things easier to works with in Kibana even though we already have the info in hf_hub
user agent)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for CLI, i addressed it directly in this PR (in f2eb79b) but yes let's do the same for other HfApi instantiations
Co-authored-by: Lucain <[email protected]>
…hub into typer-migration
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pre-approving ✔️ Feel free to ignore my last comments if annoying 😄
Let's get this merge soon! 🎉
(note: I also tested autocompletion locally which worked like a charm)
# top level single commands (defined in their respective files) | ||
app.command( | ||
name="download", | ||
help="Download files from the Hub.", | ||
)(download) | ||
app.command( | ||
name="upload", | ||
help="Upload a file or a folder to the Hub.", | ||
)(upload) | ||
app.command( | ||
name="upload-large-folder", | ||
help="Upload a large folder to the Hub. Recommended for resumable uploads.", | ||
)(upload_large_folder) | ||
app.command( | ||
name="env", | ||
help="Print information about the environment.", | ||
)(env) | ||
app.command( | ||
name="version", | ||
help="Print information about the hf version.", | ||
)(version) | ||
app.command( | ||
name="lfs-enable-largefiles", | ||
help="Configure your repository to enable upload of files > 5GB.", | ||
hidden=True, | ||
)(lfs_enable_largefiles) | ||
app.command( | ||
name="lfs-multipart-upload", | ||
help="Upload large files to the Hub.", | ||
hidden=True, | ||
)(lfs_multipart_upload) | ||
|
||
|
||
# command groups | ||
app.add_typer(auth_cli, name="auth") | ||
app.add_typer(cache_cli, name="cache") | ||
app.add_typer(repo_cli, name="repo") | ||
app.add_typer(repo_files_cli, name="repo-files") | ||
app.add_typer(jobs_cli, name="jobs") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fine for me yes! Thanks for looking into this :)
Let's get this merged once the CI is green! 🔥 |
* Refactor CLI implementation using Typer (#3365) * migrate CLI to typer * (#3364) disable rich in all cases * update tests * make typer-slim a required dep * use Annotated * fix linting issues * fix tests * refactoring * update docs * use built in types * fix mypy * call whoami directly * lint * Apply suggestions from code review Co-authored-by: Lucain <[email protected]> * import Annotated from typing * Use Enums * set verbosity globally * refactor scan cache and update version docstring * centralize where Typer is defined * no need for ... * rename enum * no need for extra param name * docstring * revert * centralize arguments and options definition * add library version when initializing HfApi * add auto-completion * sort commands alphabetically * suggestions * centralize jobs params and HfApi initialization * fix --------- Co-authored-by: Lucain <[email protected]> * update docs --------- Co-authored-by: Lucain <[email protected]>
* Refactor CLI implementation using Typer (#3365) * migrate CLI to typer * (#3364) disable rich in all cases * update tests * make typer-slim a required dep * use Annotated * fix linting issues * fix tests * refactoring * update docs * use built in types * fix mypy * call whoami directly * lint * Apply suggestions from code review Co-authored-by: Lucain <[email protected]> * import Annotated from typing * Use Enums * set verbosity globally * refactor scan cache and update version docstring * centralize where Typer is defined * no need for ... * rename enum * no need for extra param name * docstring * revert * centralize arguments and options definition * add library version when initializing HfApi * add auto-completion * sort commands alphabetically * suggestions * centralize jobs params and HfApi initialization * fix --------- Co-authored-by: Lucain <[email protected]> * update docs --------- Co-authored-by: Lucain <[email protected]>
* [1.0] Httpx migration (#3328) * first httpx integration * more migration * some fixes * download workflow should work * Fix repocard and error utils tests * fix hf-file-system * gix http utils tests * more fixes * fix some inference tests * fix test_file_download tests * async inference client * async code should be good * Define RemoteEntryFileNotFound explicitly (+some fixes) * fix async code quality * torch ok * fix hf_file_system * fix errors tests * mock * fix test_cli mock * fix commit scheduler * add fileno test * no more requests anywhere * fix test_file_download * tmp requests * Update src/huggingface_hub/utils/_http.py Co-authored-by: célina <[email protected]> * Update src/huggingface_hub/utils/_http.py Co-authored-by: célina <[email protected]> * Update src/huggingface_hub/hf_file_system.py Co-authored-by: célina <[email protected]> * not async * fix tests --------- Co-authored-by: célina <[email protected]> * Bump minimal version to Python3.9 (#3343) * Bump minimal version to Python3.9 * use built-in generics * code quality * new batch * yet another btach * fix dataclass_with_extra * fix * keep Type for strict dataclasses * fix test * Remove `HfFolder` and `InferenceAPI` classes (#3344) * Remove HfFolder * Remove InferenceAPI * more recent gradio * bump pytest * fix python 3.9? * install gradio only on python 3.10+ * fix tests * fix tests * fix * [v1.0] Remove more deprecated stuff (#3345) * remove constants.-hf_cache_home * remove smoothly_deprecate_use_auth_token * remove get_token_permission * remove update_repo_visibility * remove is_write_action arg * remove write_permission arg from login methods * new parameter skip_if_logged_in in login methods * Remove resume_download / force_filename parameters * Remove deprecated local_dir_use_symlinks parameter * Remove deprecated language, library, task, tags from list_models * Return commit URL in upload_file/upload_folder (previously url to file/folder on the Hub) * fix upload_file/upload_folder tests * smoothly_deprecate_legacy_arguments everywhere * code quality * fix tests * fix xet tests * [v1.0] Remove `Repository` class (#3346) * Remove Repository class + adapt docs * remove fr git_vs_http * bump to 1.0.0.dev0 * Remove _deprecate_positional_args on login methods (#3349) * [v1.0] Remove imports kept only for backward compatibility (#3350) * Remove imports kept only for backward compatibility * fix tests * [v1.0] Remove keras2 utilities (#3352) * Remove keras2 utilities * remove keras from init * [v1.0] Remove anything tensorflow-related + deps (#3354) * Remove anything tensorflow-related + deps * init * fix tests * fix conflicts in tests * Release: v1.0.0.rc0 * [v1.0] Update "HTTP backend" docs + `git_vs_http` guide (#3357) * HTTP configuration docs * http configuration docs * refactored git_vs_http * fix import * fix docs? * Update docs/source/en/package_reference/utilities.md Co-authored-by: célina <[email protected]> --------- Co-authored-by: célina <[email protected]> * Refactor CLI implementation using Typer (#3372) * Refactor CLI implementation using Typer (#3365) * migrate CLI to typer * (#3364) disable rich in all cases * update tests * make typer-slim a required dep * use Annotated * fix linting issues * fix tests * refactoring * update docs * use built in types * fix mypy * call whoami directly * lint * Apply suggestions from code review Co-authored-by: Lucain <[email protected]> * import Annotated from typing * Use Enums * set verbosity globally * refactor scan cache and update version docstring * centralize where Typer is defined * no need for ... * rename enum * no need for extra param name * docstring * revert * centralize arguments and options definition * add library version when initializing HfApi * add auto-completion * sort commands alphabetically * suggestions * centralize jobs params and HfApi initialization * fix --------- Co-authored-by: Lucain <[email protected]> * update docs --------- Co-authored-by: Lucain <[email protected]> * add hf repo delete command * add repo settings, repo move, repo branch commands * fix test * Apply suggestions from code review Co-authored-by: Lucain <[email protected]> --------- Co-authored-by: Lucain <[email protected]> Co-authored-by: Lucain Pouget <[email protected]>
* Refactor CLI implementation using Typer (#3365) * migrate CLI to typer * (#3364) disable rich in all cases * update tests * make typer-slim a required dep * use Annotated * fix linting issues * fix tests * refactoring * update docs * use built in types * fix mypy * call whoami directly * lint * Apply suggestions from code review Co-authored-by: Lucain <[email protected]> * import Annotated from typing * Use Enums * set verbosity globally * refactor scan cache and update version docstring * centralize where Typer is defined * no need for ... * rename enum * no need for extra param name * docstring * revert * centralize arguments and options definition * add library version when initializing HfApi * add auto-completion * sort commands alphabetically * suggestions * centralize jobs params and HfApi initialization * fix --------- Co-authored-by: Lucain <[email protected]> * update docs --------- Co-authored-by: Lucain <[email protected]>
* [1.0] Httpx migration (#3328) * first httpx integration * more migration * some fixes * download workflow should work * Fix repocard and error utils tests * fix hf-file-system * gix http utils tests * more fixes * fix some inference tests * fix test_file_download tests * async inference client * async code should be good * Define RemoteEntryFileNotFound explicitly (+some fixes) * fix async code quality * torch ok * fix hf_file_system * fix errors tests * mock * fix test_cli mock * fix commit scheduler * add fileno test * no more requests anywhere * fix test_file_download * tmp requests * Update src/huggingface_hub/utils/_http.py Co-authored-by: célina <[email protected]> * Update src/huggingface_hub/utils/_http.py Co-authored-by: célina <[email protected]> * Update src/huggingface_hub/hf_file_system.py Co-authored-by: célina <[email protected]> * not async * fix tests --------- Co-authored-by: célina <[email protected]> * Bump minimal version to Python3.9 (#3343) * Bump minimal version to Python3.9 * use built-in generics * code quality * new batch * yet another btach * fix dataclass_with_extra * fix * keep Type for strict dataclasses * fix test * Remove `HfFolder` and `InferenceAPI` classes (#3344) * Remove HfFolder * Remove InferenceAPI * more recent gradio * bump pytest * fix python 3.9? * install gradio only on python 3.10+ * fix tests * fix tests * fix * [v1.0] Remove more deprecated stuff (#3345) * remove constants.-hf_cache_home * remove smoothly_deprecate_use_auth_token * remove get_token_permission * remove update_repo_visibility * remove is_write_action arg * remove write_permission arg from login methods * new parameter skip_if_logged_in in login methods * Remove resume_download / force_filename parameters * Remove deprecated local_dir_use_symlinks parameter * Remove deprecated language, library, task, tags from list_models * Return commit URL in upload_file/upload_folder (previously url to file/folder on the Hub) * fix upload_file/upload_folder tests * smoothly_deprecate_legacy_arguments everywhere * code quality * fix tests * fix xet tests * [v1.0] Remove `Repository` class (#3346) * Remove Repository class + adapt docs * remove fr git_vs_http * bump to 1.0.0.dev0 * Remove _deprecate_positional_args on login methods (#3349) * [v1.0] Remove imports kept only for backward compatibility (#3350) * Remove imports kept only for backward compatibility * fix tests * [v1.0] Remove keras2 utilities (#3352) * Remove keras2 utilities * remove keras from init * [v1.0] Remove anything tensorflow-related + deps (#3354) * Remove anything tensorflow-related + deps * init * fix tests * fix conflicts in tests * Release: v1.0.0.rc0 * [v1.0] Update "HTTP backend" docs + `git_vs_http` guide (#3357) * HTTP configuration docs * http configuration docs * refactored git_vs_http * fix import * fix docs? * Update docs/source/en/package_reference/utilities.md Co-authored-by: célina <[email protected]> --------- Co-authored-by: célina <[email protected]> * Refactor CLI implementation using Typer (#3372) * Refactor CLI implementation using Typer (#3365) * migrate CLI to typer * (#3364) disable rich in all cases * update tests * make typer-slim a required dep * use Annotated * fix linting issues * fix tests * refactoring * update docs * use built in types * fix mypy * call whoami directly * lint * Apply suggestions from code review Co-authored-by: Lucain <[email protected]> * import Annotated from typing * Use Enums * set verbosity globally * refactor scan cache and update version docstring * centralize where Typer is defined * no need for ... * rename enum * no need for extra param name * docstring * revert * centralize arguments and options definition * add library version when initializing HfApi * add auto-completion * sort commands alphabetically * suggestions * centralize jobs params and HfApi initialization * fix --------- Co-authored-by: Lucain <[email protected]> * update docs --------- Co-authored-by: Lucain <[email protected]> * add hf repo delete command * add repo settings, repo move, repo branch commands * fix test * Apply suggestions from code review Co-authored-by: Lucain <[email protected]> --------- Co-authored-by: Lucain <[email protected]> Co-authored-by: Lucain Pouget <[email protected]>
* [1.0] Httpx migration (#3328) * first httpx integration * more migration * some fixes * download workflow should work * Fix repocard and error utils tests * fix hf-file-system * gix http utils tests * more fixes * fix some inference tests * fix test_file_download tests * async inference client * async code should be good * Define RemoteEntryFileNotFound explicitly (+some fixes) * fix async code quality * torch ok * fix hf_file_system * fix errors tests * mock * fix test_cli mock * fix commit scheduler * add fileno test * no more requests anywhere * fix test_file_download * tmp requests * Update src/huggingface_hub/utils/_http.py Co-authored-by: célina <[email protected]> * Update src/huggingface_hub/utils/_http.py Co-authored-by: célina <[email protected]> * Update src/huggingface_hub/hf_file_system.py Co-authored-by: célina <[email protected]> * not async * fix tests --------- Co-authored-by: célina <[email protected]> * Bump minimal version to Python3.9 (#3343) * Bump minimal version to Python3.9 * use built-in generics * code quality * new batch * yet another btach * fix dataclass_with_extra * fix * keep Type for strict dataclasses * fix test * Remove `HfFolder` and `InferenceAPI` classes (#3344) * Remove HfFolder * Remove InferenceAPI * more recent gradio * bump pytest * fix python 3.9? * install gradio only on python 3.10+ * fix tests * fix tests * fix * [v1.0] Remove more deprecated stuff (#3345) * remove constants.-hf_cache_home * remove smoothly_deprecate_use_auth_token * remove get_token_permission * remove update_repo_visibility * remove is_write_action arg * remove write_permission arg from login methods * new parameter skip_if_logged_in in login methods * Remove resume_download / force_filename parameters * Remove deprecated local_dir_use_symlinks parameter * Remove deprecated language, library, task, tags from list_models * Return commit URL in upload_file/upload_folder (previously url to file/folder on the Hub) * fix upload_file/upload_folder tests * smoothly_deprecate_legacy_arguments everywhere * code quality * fix tests * fix xet tests * [v1.0] Remove `Repository` class (#3346) * Remove Repository class + adapt docs * remove fr git_vs_http * bump to 1.0.0.dev0 * Remove _deprecate_positional_args on login methods (#3349) * [v1.0] Remove imports kept only for backward compatibility (#3350) * Remove imports kept only for backward compatibility * fix tests * [v1.0] Remove keras2 utilities (#3352) * Remove keras2 utilities * remove keras from init * [v1.0] Remove anything tensorflow-related + deps (#3354) * Remove anything tensorflow-related + deps * init * fix tests * fix conflicts in tests * Release: v1.0.0.rc0 * [v1.0] Update "HTTP backend" docs + `git_vs_http` guide (#3357) * HTTP configuration docs * http configuration docs * refactored git_vs_http * fix import * fix docs? * Update docs/source/en/package_reference/utilities.md Co-authored-by: célina <[email protected]> --------- Co-authored-by: célina <[email protected]> * Refactor CLI implementation using Typer (#3372) * Refactor CLI implementation using Typer (#3365) * migrate CLI to typer * (#3364) disable rich in all cases * update tests * make typer-slim a required dep * use Annotated * fix linting issues * fix tests * refactoring * update docs * use built in types * fix mypy * call whoami directly * lint * Apply suggestions from code review Co-authored-by: Lucain <[email protected]> * import Annotated from typing * Use Enums * set verbosity globally * refactor scan cache and update version docstring * centralize where Typer is defined * no need for ... * rename enum * no need for extra param name * docstring * revert * centralize arguments and options definition * add library version when initializing HfApi * add auto-completion * sort commands alphabetically * suggestions * centralize jobs params and HfApi initialization * fix --------- Co-authored-by: Lucain <[email protected]> * update docs --------- Co-authored-by: Lucain <[email protected]> * add installers * fix windows * fix log * fix workflow? * fix workflow again * add debugging steps * fix * remove bin dir and install dir params * update workflow * remove version param * document usage * [1.0] Httpx migration (#3328) * first httpx integration * more migration * some fixes * download workflow should work * Fix repocard and error utils tests * fix hf-file-system * gix http utils tests * more fixes * fix some inference tests * fix test_file_download tests * async inference client * async code should be good * Define RemoteEntryFileNotFound explicitly (+some fixes) * fix async code quality * torch ok * fix hf_file_system * fix errors tests * mock * fix test_cli mock * fix commit scheduler * add fileno test * no more requests anywhere * fix test_file_download * tmp requests * Update src/huggingface_hub/utils/_http.py Co-authored-by: célina <[email protected]> * Update src/huggingface_hub/utils/_http.py Co-authored-by: célina <[email protected]> * Update src/huggingface_hub/hf_file_system.py Co-authored-by: célina <[email protected]> * not async * fix tests --------- Co-authored-by: célina <[email protected]> * Bump minimal version to Python3.9 (#3343) * Bump minimal version to Python3.9 * use built-in generics * code quality * new batch * yet another btach * fix dataclass_with_extra * fix * keep Type for strict dataclasses * fix test * Remove `HfFolder` and `InferenceAPI` classes (#3344) * Remove HfFolder * Remove InferenceAPI * more recent gradio * bump pytest * fix python 3.9? * install gradio only on python 3.10+ * fix tests * fix tests * fix * [v1.0] Remove more deprecated stuff (#3345) * remove constants.-hf_cache_home * remove smoothly_deprecate_use_auth_token * remove get_token_permission * remove update_repo_visibility * remove is_write_action arg * remove write_permission arg from login methods * new parameter skip_if_logged_in in login methods * Remove resume_download / force_filename parameters * Remove deprecated local_dir_use_symlinks parameter * Remove deprecated language, library, task, tags from list_models * Return commit URL in upload_file/upload_folder (previously url to file/folder on the Hub) * fix upload_file/upload_folder tests * smoothly_deprecate_legacy_arguments everywhere * code quality * fix tests * fix xet tests * [v1.0] Remove `Repository` class (#3346) * Remove Repository class + adapt docs * remove fr git_vs_http * bump to 1.0.0.dev0 * Remove _deprecate_positional_args on login methods (#3349) * [v1.0] Remove imports kept only for backward compatibility (#3350) * Remove imports kept only for backward compatibility * fix tests * [v1.0] Remove keras2 utilities (#3352) * Remove keras2 utilities * remove keras from init * [v1.0] Remove anything tensorflow-related + deps (#3354) * Remove anything tensorflow-related + deps * init * fix tests * fix conflicts in tests * Release: v1.0.0.rc0 * [v1.0] Update "HTTP backend" docs + `git_vs_http` guide (#3357) * HTTP configuration docs * http configuration docs * refactored git_vs_http * fix import * fix docs? * Update docs/source/en/package_reference/utilities.md Co-authored-by: célina <[email protected]> --------- Co-authored-by: célina <[email protected]> * Refactor CLI implementation using Typer (#3372) * Refactor CLI implementation using Typer (#3365) * migrate CLI to typer * (#3364) disable rich in all cases * update tests * make typer-slim a required dep * use Annotated * fix linting issues * fix tests * refactoring * update docs * use built in types * fix mypy * call whoami directly * lint * Apply suggestions from code review Co-authored-by: Lucain <[email protected]> * import Annotated from typing * Use Enums * set verbosity globally * refactor scan cache and update version docstring * centralize where Typer is defined * no need for ... * rename enum * no need for extra param name * docstring * revert * centralize arguments and options definition * add library version when initializing HfApi * add auto-completion * sort commands alphabetically * suggestions * centralize jobs params and HfApi initialization * fix --------- Co-authored-by: Lucain <[email protected]> * update docs --------- Co-authored-by: Lucain <[email protected]> * Make HfHubHTTPError inherit from OSError (#3387) * Release: v1.0.0.rc1 * print relevant message based on the linux distro * better warning message * log info instead of warning * copilot suggestions * Update utils/installers/install.sh Co-authored-by: Copilot <[email protected]> * update docs --------- Co-authored-by: Lucain <[email protected]> Co-authored-by: Lucain Pouget <[email protected]> Co-authored-by: Copilot <[email protected]>
Resolves #3363 and #3364.
This PR migrates the
hf
CLI from built-inargparse
toTyper
using idiomatic, function based commands. this significantly reduces boilerplate, making the CLI easier to maintain, less error prone, and faster to extend with newHfApi
features.What changed
new dependencies:
typer
, latest version is 0.17.4, it should be stable enough and it's widely adopted.rich
installed or not.Main entrypoint:
cli/hf.py
now uses typer.Typer and mounts:download
,upload
,upload-large-folder
,env
,version
(all defined in their respective files).auth
,cache
,repo
(withtag
subgroup),repo-files
,jobs
(withscheduled
anduv
subgroups).lfs-enable-largefiles
,lfs-multipart-upload
.tests/test_cli.py
has been updated accordingly. cf the documentation to write tests for typer cli: https://typer.tiangolo.com/tutorial/testing/#about-the-appcommand-decorator.